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ABSTRACT 

The DX SYSTEM of Diagnositic Testing is an easy-to-use 
computerized system for developing aQd administering diagnostic 
tests. Avdiagnostic test measures a student's mastery of a specific 
domain (skill or content area). It examines the necessary subskills 
hierarchically from the most to the least complex. The DX SYSTEM 
features tailored testing with immediate student feedback. In 
addition teachers can obtain an immediate student performance 
profile, v*'' us additional information to help diagnose and remediate a 
student's oblems. This manual provides detailed directions' for 
operating tne DX SYSTEM on an Apple II computer. The steps include: 
(1) "booting up"; (2) making a back-up copy; (3) the 4nenu; (4) hov^ to 
enter test items; (5) administering the test; and (6) how to obtain 
test results for individual students. In addition it describes six 
steps for developing diagnostic tests and test items: (1) developing 
a blueprint of the domain; (2) mapping prerequisite skills for the 
target domain; (3) developing a .hierarchy or sequence of skills; (4) 

jwriting test items using common conventions of item writing; (5) 

reviewing the test items; and (6) field testing the items. (BS) 
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DX, More than a Computerized Testing System 

The" diagnostic testing system Is, first and foremost, a system for 
diagnosing problems of iiyli vidua! students In a particular skill or content 
area. For convenience, we will call th^ target skill /content area a domain 
throughout the text.- The critical ffiatures^of this system are: 

1. Each test Is tailored to the Individual student. Because the test is 

I 

• administered 'one on- one', using a computer, the test taker's pattern 

»• • • 

of responses determines what kinds of items, and Ftow many questions 

4 

s/he will receive in the assessment of a particular doma>n. ^ 

2. Each student's performance within a 'domain,; s diagnosed by assessing 
the skill as a whole, and by assessing the prerequisite subskills, when 
necessary. 

0 

3. Each student receives immediate feedback after each response. Thus the 
testing event is also a learning occasion., . 

4. Because the test is computerized, the teacher can obtain a performance 
profile of each student Immediately after the testing event, if the 

. teacher so chooses. 

' 5. Because of the structure of this test, the teacher can obtain several 
kinds of information which may help him/her to diagnose and remediate a 
student's problems. 

r' 

6. The DX system is gasy to use because the instructions foy its 

operations are simple, and require no knowledge of specialized computer 
languages. • * 
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This manual will give you a step by step description of how to use the 

s 

DX system for .several purposes such as: writing your own test Items; 
. administering *the test; obtaining test results. Before starting the 
'operations^ section of the manual , however, we strongly urge you to read the 
' next section^ "A diagnostic model of testing." This section describes more 
fully how the system diagnoses students' strengths and weaknesses. 



/ 
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• . The Diagnostic Testing System 

What the test measures • / 

Thg goal of .the diagnostic test is to measure induidual students' 
mastery of a well specified domain. The assessment instrument examines the 
critical and prerequisite subskills or subconcepts which the student needs 
in order to piaster the domain. These skills should be arranged, - 
hierarchically from the"^inbst complex down to the least complex. If a 
student clearly has not mastered the domain, the system will administer 
items from domain subskills until it finds the student's maximum level of 
performance within t^ hierarchy. The testing system is -designed so that 
the target domain and each of its ><;e requisite skills/concepts is measured 
by a cluster off five items. These itdm clusters are hierarchically 
arranged within the DX system such that^ach cluster represents a decision 
point "for what item the test taker is s^sequently given. (See figure 1) . 
The hierarchy of clusters can be arrange^l in two to four levels. To 
progress along these levels, the students must make errors in two out of 
five items within a oa>uster. Mastery of a skill is indicated only Jf the 
student succeeds Jn four out of five items of 'the most complex skill 
cluster. The test stops when the test taker succeeds at a complete item 
' cluster, of when s/he reaches the bottom of the hiei-archy. At either point 
it is assumed that the individual has illustrated his or her degree of 
•mastery with regar^d to the target skill or content area. 

An additional feature of the^ cluster hierarchy is that the distractors 
for all items across all levels of the hierarchy fall .within one to a 
maximum of four error categories. (See figure ^) These error categories 
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Figure 1 

Sample Hierarchy of Items for the Domain of Inference 



c 



Cluster #Y 

Target Domain: . 
Making inferences 
from a passage 



Cluster #IV 



subskill: making 
inferences frotn 
two sentences^ ) ^ 



IV 3 

V" - 



Cluster # III 

* * 

subskm : integrating 
literal information 
from two sentences ' 



IU^=~^ 3 



i 



Cluster #11 

subskill: integrating 
and identifying 
information from one 
sentence 



Cluster #1 » 

subskill:. identifying 
vocabulary and content 
from one sentence 

' ^ 



I 



1 - 5 are individual question numbers. 
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Figure 2 

Sample Dlstractor Categories for Inference Skill 

Example 1 Item 1 B 

A 

Dlstractor Categories Correct 

• B Item 4 Correct.- 

A: False but plausible ^' \ 

Information ^ " J 

' Item 2 A ' . B 

B: Information that Cluster B A 

answers only part V B „ • ----- 

of the question • A item 5 A • 

Correct „ B 
B 

' Item 3 A A 

ty A Correct 

Correct 
B 

B . 



ExaiHple 2 

Dlstractor Categories 

A: False but plausible 
information 

B: Information that 
answers only part 
of the question 

C: Information which is 
completely irrelevant 
to the question 

1 



CI us.ter 



item 1 A 
C 
«.A 
Correct 

B 



Item 2 C 
A 
B 

Correct 
A 



item 3 A 

' Correct 
B 

B • 
C 



item 4 C 
A 

Correct 

B 

A 



item 5 C 
C 
A 

Correct 
B 



I ■ 
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represent the teacher' | hypotheses about the kinds of misunderstandings the 
test taker has with respect to the domain or the prerequisite skills. 

An Example ' ' > ^ ^ 

Let's take a concrete example of how this test works (See figures 3 
and 4). The domain we wish to assess is the student's ability to make 
infereoces. The totality of this skill is measured by cluster #5, 
illustrated in figures 3 and 4. If the student errs in. two out of ..the five 
'items in this cluster, s/he will be presented with items from cluster #4, 
and so forth. Note that the student will be presented with items from a 
cluster until s/he makes two errors. Thus, an individual may successfully^ 
complete items 1-3, but err on items 4 and 5, then she will be presented 
with the next cluster. Or, the test taker may err on items 1 and 3, and 
then receive items from the next cluster. 

Student #1 of .the illustration (figure 3) successfully made infereijces 
from small chunks of text (such as two sentences), but had difficulty 
integrating and making inferences from larger text chunks such as a 
paragraph. Student #2, (figure 4) on the other. hand, had trouble 
identifying and integrating Ijteral information from even one sentence. 
Yet we know that vocabulary (cluster #1) is not the problem. 

At , this point the user^mayVant to hypothesize the yeasons for a 
student's problem with the skill area. One helpful tool may be to look at 
the , test talcer's distractor patterns throughout the test. Notice that 
student #2 (figure 4) of the illustration seemed to consistently make the 
same type of error throughout the item clusters in the hierarchy. That is 
s/he consistently selected the distractor falling in category A of our 
example: 'plausible but false information', thus one potential aspect of 
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Figure 3 

How the Item Clusters and Response Categories Fit Together 
and Example/of #1 Test Taker's Path Through the Test 



CLUSTER 



ITEM NUMBER 



EXAMPLE 
RESPONSE SELECTION 



Target DomaTh: 
Making Inferences 
from a passage 



1 
2 
3 

5 



IV 

Subskill making 
inferences from 
two sentences 



1 
2 
3 
4 
5 



B, wrong choice — 
A, wrong choice - 



correct 



, correct 
not given 



correct 



correct 




"A, wrong choice 

4 

correct 




correct 




Stop testing: - Advise more 
instruction of subskill IV 



4 



11 
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F1 gure 4 
ExaiTiple of #2 Test Taker's Path 
/' Through, the Test 



CLUSTER 



ITEM NUMBER 



EXAMPEE 
RESPONSE SELECTION 



Target domain: * 
Making ^inferences 
from a passage' 



IV 

subskill: Making 
inferenries from 
two sentences 



1 

'2 
3 
4 
5 



1 
2 
3 
4 
5 



III 1 
^subskilT: Integra- 2 
*t4ng liternal infor-3 
mation from tj^o 4 
sentences 5 



ir " 1 
subskiH: Integra- 2 
ting and identify- 3 
ing information 4 
from one senence 5 



I '1 

Subski 11: Identify 2 

ing vocabulary and 3 

content from one 4 

sentence 5 



A, wrong choice— —> 
A, wrong choice .^^^ 
not given 
not given 
not given 




A, wrong choice-~p 

B, wrong choice.<S£r 
not given 
not given 
not given 





correct 

A, wrong choice 
correct 

A, wrong choice^ 
not giv^n ' 



correct ''^^^^ 
correct -r 
A, wrong choice'! 
cocrect 

A, wrong choice 



correct 
correct 
correct 
correct 
correct 




Stop testing: advise 
instruction in sub- 
skill II with atten- 
tion to plausible in- 
fownation. 
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Instructional remediation may be to teactAhe student to distinguish 
plausible but false 'information from information which is clearly confirmed 
by the paragraph. * 

Student # 1 was Successful in making inferences from two or more 
sentences. This student, however, seemed to select distractors which only 
— partially answer the question (B). It's>possible tJjat the student is 
either misinterpreting the question (a test taking skill) or that s/he has 

jt 

not learned "to integrate large chunks of text (an inference skill). In 
this case, the teacher may want to further test the student on this 
prerequisite skill. 

\<'' 
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HOW TO OPERATE THE. 
DX SYSTEM 
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The following pages will show you how to us^' the DX 
package. 5ust follow the steps described below. You will 
need the following etjuipment: ^ 

- an Apple II computer with'a monitor (TV screen), 
•two disk drives and a 48K memory 

- the DX disk which includes some sample items 

- blank disk 

- a printer would be handy but is optional. 

If you are already familiar with the use of the Apple 
II computer and the PASCAL language, make a back-up copy of 
the program disk and store it in a safe place. Then, go to 
Step 3, "The Menu" on page 14. 

. If you are not familiar with the use of an Apple II 
computer or with the Pascal language*, be sure to read the 
following pages and follow all the steps very closely. 

Computer orientatiy* 

What do these computer components do? 
Thk actual computer is the box on which the video 
screen is sitting. That box Is filled with microchips 
which store information. The computer also brings 
information which has been programmed Into it. This 
information tells the microcomputer hqw to interpret 
and store the information you and your students put 
Into it. * 



* For a detailed description of how your computer and how 
it works please consult your Apple II manual. 

15 



The control card is simply a card containing 
microchips with special instructions on 1t. For 
example, 'you may want your screen to provide 80 
columns of typing space instead of 40. The^ 
machine would need a particular control card^ 
inserted in order to do this. YOU DO NOT NEED TO 
MANIPULATE A CONTROL CARD. JUST MAKE SURE THAT 
THE COMPUTER HAS ONE BEFORE YOU START. The 
control cad should have been placed in the 
computer when it- was set up. 
The disk drive is like the starter on a car. You need 
It to start, the computer. The "key" is the disk, in this 
case, you would need^a DX disk . (The DX disk is also called 
a program disk.) The DX disk carxies information which has 
been programmed into it as well as instruction for the 
interpretation and storage of new information. The disk 
"tells" the computer what to do. It also provides the 
information which you set on the screen. The disk also asks ^^ 
you what you want to do. Some computer come with 1 disk 
drive, others with 2 disk drives. You will see the 
importance of this in section 2 - "Making a back-up copy." 

9 
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Step 1 - "Booting up" 

When you Insert the disk Into .the disk drive, you are 
"booting up" the disk. This Is like putting the key into 
the starter and turning the car on. Here's how to do this: 
* a) Pick up the DX disk by holding It by the end with > 
the label, labelled side up. Don't let your fingers 
touch the shiny plastic surfaces which show through, 
the slots. The natural human oni^&ou1d cause 
problems with the machine. 

b) Insert the disk Into the disk drV'e. Some drives 
may have a closing mechanism of some sort that you 
have to operate after inserting the disk. ♦ 

c) Turn on the computer screen by using the switch on 
the rear left of your Apple II computer. Then turn 
on the video monitor as you would turn on a 
televislQri. 

d) At this point you should hear, a whirring sound 
coming from the disk drive. Also, you may see a 
light turn on indicating that the computer is 
operating. At this point, a' Menu will appear on the 

^ s.creen. You are now ready to begin the next phase 

of orientation. 
A WORD OF CAUTION: Nsver* remove or insert a disk into 
a drive that Is still in the operating state. This would 
cause severe damage. 

17 



" 15 - 



this p^fnt, the manual will ask you to type eertain 
i 

things, press certain keys, and will refer to what appear 
on the key. For your convenience, all typing instructions 
win appear in the left hand column and all screen illustra 
tlons will appear on 'the right. 

* • 
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Stfp 2 - Maklnqi^a Back-Up Copy 



A "back-up" copy of the.DX disk is sijnply an extra, 
emergency copy of the disk. «You will use the copy to wa,rk 
from. You must store the original DX disk in a safe place, 
away from heat and moisture, tt is iinportant to make a copy 
of the original disk in tase there is an unexpected problem 
with the systej^ or the copy is damanged. Make the copy as 



1. Insert the DX disk into the n drive. Insert the . 
copy disk into the #2 drive. Remember, hold the 
disk by the Table, with the Table side up. 

9 

2. Turn on the computer and the monitor. Wait for the 
whirring sound and the operating Tight. 

^ To make a backup copy: 



foTTows: 



YOU TYPE 



SCREEN SH(^WS 



Hi , WeTcome to the DX 
system. What wouTd 
you Tike to do? 



COPY (press return -key) 




19 



- 17 - 



Before going m further, store the original disk, (the program disk 
supplied with this manual). Fron, now on. you will use the back-up copy you 
just made. 

Now that you have your back-up copy, you are reatly to see what the DX 
system Is all about. Place your DX disk in Drive 1. 



20 
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Step 3 - The Menu 

To begin by getting the menu: 



YOU TYPE 



SCREEN SHOWS 



SEE MENU (press return key) 



) 



H11 Welcome to the 
system. What would 
you like to do? 



DX 



MENU 

W Write new ^est items 

R Revise existing items 

A Administer the test 

G Get test results 

S Directory of sudents 

D Directory of test Items 

Q Quit 



(Type 1n letter Indicating what you would like to do. See subsequent 
sections under the Menu headings for specific instructions). 

* Press the return key each time you type In a command. 



21 
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Step 4 - How to Enter the Test Items . 

This section viHT Illustrate how to enter the test items, including 
the passage, item stem, questions, response choices and answer key. For 
information about how to develop effective test items, please see section 

There are a few important items to keep in mind as you enter the test 
items. First, remember that you have a limited space for entering the item 
passage, qjie^tions, response choices and answer key. This space consists 
of 80 columns (or spaces) across and 24 lines down the screen. The column 

• space includes space fo^ punctuation mark:s and spacing between words. 

■t 

Second, you have to label each passage and question set with a cluster 
label. These cluster labels will enable the computer to select particular 
«o* terns and questions for analysis and will help you to sort and access this 
information when you need it. 

> 

4 
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W "Entering New Test Items 



YOU TYPE 



SCREEN SHOWS ' 



W 



{Give a skill cluster name) 
OVERALL INFERENCE 

(Indicate number of questions) 

, 2 

(Give a name for each distractor 
category) 

A « FALSE B"T PLAUSIBLE INFORMATION 
B « INFQP riON THAT ANSWERS ONLY 
PART OF THE INFORMATION 



(An asterisk lets the computer 
know that you have finished v 
entering the test passage.) 



' What is the' name of this skill 
cluster? 

\ 

.How many questions will you give 
this skill cluste. 

What 'fire the distractor 
categories? 



Begin writing the text 
passage here. End\with 
asterisk.^ 



an 



Many creatures of the world 
change compl^etely as they grow 
from babies to adults. This 
complete change of form is 
called a metamorphosis. Baby 
frogs are a good example of 
metamorphosis. 

•Adult frogs lay eggs. 
These eggs change into sn,:lt 
fish-like creatures called 
polliwogs. They have no limbs. 
Their bodies change and they 
become frogs. ^When the frogs 
lay eggs, the process will begin 
again.* 

(Example test passage) 

Write question 1 here, including 
4 answer choices. End with an 
asterisk * followed by 
correspondence between choice 
and disti^ACtor categories. 
Example: 

*1«A, 2*B, 3 » correct, 4«C 
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* • YOU TYPE 



SCREEN SHOWS 



(Type in questton and answer 
choices) 

THE MAIN IDEA OF THIS PASSAGE IS: 

1 Many creatures lay egg^. 

2 Many creatures become fish 
as they grow. 

3 Many poUiwogs lay eggs. 

4 Many creatures undergo • 
metamorphosis as they grow. 

*1«A, 2=A, 3-B, 4»correct. 



(Type in questi6n) ' 



Write question 2 here, \ncluding 
4 answer choices. End with an 
asterisk * followed by 
correspondence between choice 
and distractor categories. 



This passage is mosMy about: 

1. whatnpolliwogs look like 

2. how frogs grow from eggs 

3. what fish look like 

4. how polHwogs grow legs 

*l=a, 2=c»rrect, 3=A, 4-B 

End of skin cluster, "Overall 
Inference". We are saving your 
test and questions on disk. 

Now write the text for the next 
question. Remember that you 
have 3 questions left. 



24 
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YOU TYPE 



R REVISE EXISTING TEST ITEMS 

SCREEN SHOWS 



■ \ 



(enter cluster name) 
Overall Inferences 

« 

(To retype, delete, 6r correct 
words and puntuatlon marks, use 
the arrow keys andvtype In your 
corrections. To move to the 
next line, press the RETURN key.) 



(Now type any of the foil owl 



/ 



U (to finish and update* the disk 
file with the latest changes) 



R (,to return to. revising this 
cluster again) 

E (to exit** without perserving 
any changes.) 



Which test cluster would you 
like to edit? \ 



Test Passage 1 

Many creatures of the world 
change completely as they grow 
from babies to adults. This 
complete change of form Is 
called a metamorphosis. Baby 
frogs are a good example of 
metamorphosis. 
, ^ Adult frogs lay eggs. 
Thes'^ eggs change into small 
fish-like creatures called 
polliwogs. TKey have no limbs. 
Their bodies change and they 
become frogs. When the frogs 
"^ay eggs,, the process will begirj 
again.* 

Many creatures lay -legs. 

Many creatucs lay (use arrow key 

and correct) 

Many creatures lay eggs. 
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To "update" means to permanently replace, correct or, -change what 16 
on the disk. By permanent we mean that the next time you use the disk, 
the changes you have made will appear. , If "you do not want to preserve 
the changes^ then press E for exit. • - . . 



By exit, we^ean that you do not want to continue making changes in 
this particular cluster. You may, at this point, make changes In 
another* cluster, or exit the program altogether.-^ 25 



step 5 - Admlnlsterlny the Test 
,YOU TYPE . / 



SCREEN SHOWS' 



(first and ld§t name, eg:) A 
"Bonnie Brooks 



Welcome! ^Please ^fer 'yotir 
first and last name. 



HI Bonnie. I will .show^you 
a passage and a question 
abbut it. Each question 
needs just one answer. 
When you read the question, 
give your^answer by 
pressing JJ^ or B or C or D. 



(type in your response) 
A , 



/ TTa 



your chot^? 
Y/N 



Now type S (for start) 
(test begins) 



The name the student types will be used by the computer as the label 

■» 

under which his/her test results will be stored. "Whenever you try to 
obtain these part1cul,ar results, you must give the computer the name just 
as the studen.t typed it in — Including any spaces and punctuation marks. 
The test taker could have given any name, such as "banana'\ and the computer 
would accept it. It does not really matter hov/ a test taker enters his or 
her name, so long as you are aware of the name being used.. Later, you can 
use the same nam? for access to that student's results. So, if an examinee 

types in gibberish, then you may have trouble in gaining access to that 

* .„ ■ • • . 

student's file. 

If the test taker accidentally or intentionally presses any other key 
on the keyboard other than A, B, C, or D, the "What Is your choice?" 
question will be reprinted,' including the four alternative key choices. . 

26 
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Step 6 - How to Obtain the Test Results for Individual Students 

- While thij student has been taking the test, the computer has been 

recording the response patterns; and preparing diagnostic Interpretations. 

In our example test the computer recorded the following 

The number of correct and Incorrect response at each 
cluster level . 

A total for all of the clusters presented to the test 
taker. 

The number of times the test taker selected distractors 
from each distractor category. 

To obtain and read the test results for each student, you must return 

to the Menu. After the farewell greeting which signalled the end of the 

test, the Apple II cursor appeared at the bottom of the screen. Now 

t. 

YOU TYPE SCREEN SHOWS 



RUN MENU 
(pause) 

Press return and wait ' RUN MENU 

G Whose results do you want? 

The results are stored as a file named for the person tested. To obtain a 
particular file, do the following: 
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(Type Test Takers Name) 
Bonnie Brooks 



SAMPLE TEST 1 
CI uster Items Wrong D1 stractors 
Correct 
V 3 2 AB 

IV 3 2 BB 

III 4 1 B 

Advice: Bonnie needs more work at 
clusters IV and Y. D1 stractors 
suggest she needs to-pay attent4ofh 
to ,B(Descr1be B hypothesis) 

Do you want printed copy? 

(Y?li) 



The computer wants to know whether you want the test results printed. If 
you say N (no), the screen will just show you the file. If you say Y 
(yes), both the screen and the printer will give you the file. Let's assume 
that you want a printed copy: 



...Make sure your printer is ready, 



To make sure that you are ready to print, check the following: 

1. Is your printer connected to the computer? 

2. Have you switched the printer on 

3. Have you switched on the "on line" button located on front surface 
of the printer? . - 

4. Does the pointer have sufficient paper and ribbon in it? 
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HOW TO DEVELOP THE DIAGNOSTIC 
TEST AND TEST ITEMS 
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How to Develop the 
Diagnostic Test and Test Items 

The following section Is Intended to provide guidelines for the development 
of a good diagnostic test structure and good diagnostic test Items. It Is 
by no means the only way Of developing tests nor does It represent an over- 
view of test development methodologies. 

How does one go about building a good diagnostic test? 
The followlnjg five steps can guide this process: 

I. Develop a blueprint of the skill or content are a (to be labeled 
'domain' for the remainder of this section) you want to diagnose . For 
example, If you want to develop a test for reading comprehension, what are 
the major skills that lead to good reading comprehens4on? • How can you tell 
whether someone Is a good comprehen^er? Specify this as clearly as 
possible. 

II. Develop a map which desert' bes the tasks and subtasks that are pre^ 
requisite to the target domain . For example, a prerequisite skill which 
allows readers to develop comprehension skills is the ability to decode 
chunks of text rapidly and accurately. Specify as many such important pre- 
requisite skills as possible. 

III. Develop a hierarchy or a sequence of skills going from the pre- 
requlste skills or content for the domain, up through the critical skills 

« 

of the domain . This will help you determine in developing hypotheses about 
the student's overall strengths and weaknesses by their performance on 
groups of items.. It will also be a critical component In examining the 
test takers' error paterns. 
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IV. Write the test Hems based on the blueprint and prerequisite skin 
maps using common conventions of Item writing . 

V Review the test Items to confirm their match to the Item speclflca- 

""" ■ ■ ■ •" ■ ■■ ^ 

tlons and'to assure that Items do not contain extraneous complexities, 

unintended cues, or other technical flaws. Ideally, persons other than thi 

test developer would conduct this review with a 'cold eye . 

Yi. Field test the Items to determine where Item revisions are necessary, 
to clarify oY revise the relationships - between the critical sknis and 
the prerequisite skills, and to determine the number of It^ms required for 
testing. 

Each of these steps Is described In the following sections. 
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STEP I: DEVELOPING A BLUEPRINT OF THE DOMAIN 

This step Involves Identifying and specifying the domain you wish to 
diagnose; Including what 1,ts critical skills and concepts (or knowledge 

• #■ 

4 

base) are. The following questions serve as guides In formulating the 
target domain: ' 



1. How much, instructional time does it take to teach the domain? You'll 

want to i^elect a domain which covers a reasonable amount of 

. ■ ' (■ 

instructional time. For example, if a sciencfe curriculum consists of 
four major areas (i.e. biology, chemistry, physics, astronoiny) and each 
of these is in turn divided Into several subcomponents {i.e. biology; 

botany, ecology, etc.) it may be more efficient to develop a diagnostic 
f , — 

test for each of the four major areas, but not for the area 
subcomponents, or for all four areas' at once. 

« 

2. How does the objective relate to other hlgher-order^skill s? Be sure 
that the knowledge and skills you are testing and teaching reflect or 
are prerequisite to higher level thinking and problem solving skills. 
If we take the science example from above, the curriculum probWly 
stresses the understanding and application of the scientific modeller 
problem solving. Your test should examine the students' knowledge of ; 
this higher level, skill. To examine factual knowledge without 
integrating it with thinking skills would result in an trivial 
exercise, not conducive to' the betterment of instructional interven- 
tion. 
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3. How does the domain relate to long term currlcular goalsr As mentioned 
previously, your target domain should reflect an important part of your 
curriculum. However, all curricula have both short and long term goals. 
In our example science curriculum, the four areas and their subcomponents 
constitute definite time slots of .instruction, units within the texts,* 
-^tcv Yet t he -ski^t-of-ftpp^ytng-^ sc i ence model f oi^ probl em-soljdii g I s an 
overarching skill which crops up throughout the entire science curriculum. 
Thus you could develop a •tttar^osttc science tests which holds this thinking 
skill as its target skill or domain, but which determines to what extent a 
student can use this skill across or within the four science areas. 

To specify the domain then, you need to identify a critical part of the 
curriculum ^r a critical overarching skill or concept that would be 
fruitful to assess. One approach to doing this is to parse the entire 
semester's continuum for a content area (i.e. science or reading) into 
testable units. For example, you could take the hypothetical science 
continuum described above and examine each of the four areas, making sure 
to test students' progr^s using the scientific model of problem solving. 
Using this approach, you might ^ee how easily students apply the model 
within and across the four major content areas. 

Another Important consideration is the level of cognitive complexity 
students are expected to function. For example, following Bloom (1956), 
does the skill of interest involve recall, application, analysis, 
synthesis? Or. following Gagne (197 ), dees the skill represent concept 
learning (concrete or abstract), prIncipVes, procedures, problem-solving? 
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After completing step one, you should have a clear vision of the domain you 
wish to test, Including Its main components and boundaries.. Let's use as ' 
an example the area of reading comprehension. Figure 5 Illustrates a 
'portion of a sample continuum for reading comprehension. Notice that the 
content of the passage is parsed Into:' science, social studies, math, 
reading and life skills. There are alsosome core skills which are 



emphasized throughout the units such as: inferring the main idea and 

Identifying supporting details. Using such a continuum, our R>pothetica1 

test developers could specify their blueprint domain as follows: 

Domain: The learner win analyze a given narrative or factual passage and 
correctly infer the following aspects; 

♦ 

1. main idea . 

2. concepts or relationsips which support the main idea. 



V 



34 



ERIC 



< 



Figure 5 
Sample Reading Cont;inuum 



Time Unit 
Frame 



Sept./ 
Oct. 



Comprehension 



Skin 



Content Area/Life Skill 



Decoding 



Main Idea 
Comparison 
Supporting Details 
Referents 
Other Details 



Social Studies Suffixes 

Narrative: fables, moral tales 

Resources: Government Informa- 
tion Pamphlets 



-ive, -leal, 
-al 



Vocabulary 



Word identification 
Vocabulary develop- 
ment 

(Synonyms/antonyms) 
(multiple meanings) 



Oct./ 
Nov. 



Character 
Main Idea 
Supporting Details 
Comparison 



Science Reading 

Narrative: Biographic Styles 
Resources: Emergency Skill 
Pamphlets 



Prefixes anti-, pro- 
Suffixes -tiy, -ure 



Word identification 
Vocabulary develop- 
ment 

(classification) 
( synonyms/antonyms ) 



CO 



Nov./ 3 
Jan. 



Sequence 
Main Idea 
Supportlfig Details 
Character 



Math Reading 

Narrative: Biographic Styles 
Resources: Assembly or Opera- 
ting Instructions 



Prefixes inter-, intra-, Word identification 
extra- VocabuUry devel op- 

Roots rupt, duct, ment 

mit (maltiple meanings) 

' (classification) 



Jan./ 4 
Feb. 



Referents 
Cause/Effect 
Sequence 
Main Idea 
Supporting Details 



Science Reading 
Science Fiction 
Resources: Application 
Directions 



Roots diet, ject, 
tract 

Combining 

Forms Centi, kilo 



Word identlfictition 
Vocabulary devjelop- 
ment 

(multiple msanings) 
(context clues) 
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ST^P 11: MAP OF PREREQUISITE .SKILLS FOR THE TARGET DOMAIN 

Notice that the continuum also Indicates a number of prerequisite 
skins for the domain described previously. These could be specified as 

follows: V 
Prerequisite skills: In order to successfully master the domain or 

. inference, the learner must have the following 
prerequisite skills: ^ 

1. Speedy and accurate decoding skills . In addition to accurately 

• decoding words and groups of words, the reader must also recognize word 
parts which appear frequently and which Indicate meaning such as 
suffixes, prefixes and roots. 

2, The reader must have adequate knowledge of passage vocabul ary and 
skills for Identifying vocabulary meaning from context clues." 
(See Figure 6) » 

- .' ) ' 

* , t 

/ 
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STEP III: DEV|LOPING A HIERARCHY OR SEQUENCE OF SKILLS 
Now that our hypothetical test development team has blueprinted the domain 
and mapped out the prerequisites, they need to plan either a hierarchy or a 
sequence of them. Why? There are several reasons for this. First, the 
purpose if a diagnostfc test is not simply to identify where a student's 
problems are. A really useful diagnostic tool should provide some insight 
as to the kin^ of misunderstandings or skill^gaps^irTtuderTtnia^^^ 
building a test^-structure around a skill hierarchy or sequence, the test 
developer can developi better hypotheses about the kinds of problems 
st]idents may have. This would- lead to" better instructional remediation. A 
second reason for specifying the f.kill hierarchy is that it should guide 
the computer's test administration. Recall that this test is administered 

KM 

one on one by the computer. The computer must decide what item(s) to 
present the test taker based on the test taker's response. If the test 
taker starts the test with items from the most complex portion of the 
hierarchy— and fails— then s/he needs to be presented with items from less 
complex skill areas and so forth, until s/he reaches his or her maximum 
level of expertise. A student could indeed reach the "bottom" of the 
hierarchy. The test developer needs to provide the computer with a roadmap 
for doing this. Figures 6 and 7 which illustrate the domain bluepri^^t and 
prerequisite skill map also show a suggested sequence of test items from 
each of the cells. ^ 

/ 

* 
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EXAMPLE: / * ^ 

Supose the teacher wants to diagnose students' Inference skills for 
both narrative and scientific passages. A student could progress through 
the domain hierarchy as follows: 

la, lb 
2a, 2b 
3a, 3c 

If -the teaiJher Is Interested orfly In diagnosing perforinance-i«^1th one type 
of text de.' narrative) then trje path could be: la, 2a, 3a and so forth. 

* 

« \ 



/ 
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Figure 6 
Example Prerequisite Skill Map 



Narrative 
Passages 



Science 
Passagts_. 



— 

decoding 


prefixes 






suffixes 




• 


roots 






whole words 






vocabulary 


recognition 






context cues 
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Figure 7 
Example Domain Blueprint 



Main 
Idea 


4 

la 


lb 


Supporting 
'concept or 
relationships 


2a 


2b 


Details which 
strengthen 
the main Idea 
or support 


t 

3a 


3b 
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STEP IV ■ WRITING TEST ITEMS USING COMMON CONVENTIONS OF ITEM WRITING 

There are a variety of formats that can be followed for Item writing. 

The one described below combines models suggested by Popham (1980) and by 

Baker (1976) and Includes t^e following components; 

General description - a brief description of the 
objective, skill, or knowledge to be measured. 

Sample Item - a model of what test Items are to look 
like, including directions to be given to students. 

Content limHs - a description of the nature of the 
question that is to be presented to students. 

Respo'^se limits - a description of the response options 
provided to students for selected response Items or for 
constructed respqnse items the set of rules or criteria 
* that are to be used to judge the quality of a students 
' ' response. 

The first two components are relatively straightforward: a statement 
of the objective selected for testing and Instruction and the sample item 
that has been devised for assessing It. Included here also the directions 
that will be given to students. Explicit attention to the directions early 
on helps to assure that they will be clear and that students will 
understand how to complete each item. Specifying content and response 
limits/Is more complicated and is explained in greater detail below. 

Content limits describe the range of eligible content from which test 
Items may beVltten. They may include rules for creating questions, rules 
for Inclusion of prompts, cues, or additional materials (e.g., pictures, 
graphs, reading selections); 

/ 
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Content limits for selected reyonse Items define and restrict the 
"characteristics, format, and eligible content and rules to be included In 
the Item stem. By systematically Including the different situations; con- 
texts In which the skills are to be applied and/or the rules which define 
the assessed skill, test Items can provide valuable diagnostic Information, 
e.g.. In what situations are students able to demonstrate a particular 
skill, what rules have students mastered? For instance^ for a multiple 
choice Item assessing students' skill In using appropriate ^rx)nouns, the 
content limits might be as follows: The Item will present the student with 
a short (3-5 sentence) paragraph describing an action or event Involving 
two or more named individual s. 

A blank will replace the named Individual (s) 1n one 
sentence. 

Students will be asked to Identify the pronoun which 
correctly completes the sentence. 

> ■ . • 

Items will be written to exemplify the following rules: 

When the pronouj is the subject of a sentence or clause 
it should t>e in the nominative case. 

When the pronoun is the indirect object ft should be in 
the objective case. 

Such a systematic plan enables a test to diagnose whicfr rule(s) are 
causing students' difficulty; the problem ascertaining the number of items 
to be written to reflect*each rule Is addressed in a late section. 

Response limits provide rules for generating the correct response and 
Incorrect alternatives for selected response questions and rules and 
criteria for Judging the quality or correctness of a student's constructed 
response. Like the content limitf, response limits help define the range 

43 
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Of eligible content but here the focus 1s on student responses: what 
discriminations are expected and reasonable; what are the characteristics 
of an acceptable response. For selected response items, response limits 
provide rules for constructing the correct answer and the distractors, or 
wrong* answer alternatives for each Item. These rules should assure 
distractors that represent common student errors and which thus may provide 
Important diagnostic information. For example, response limits for the 
pronoun example described above might be as follows: 

Four alternatives will be provided for each Item, the 
correct answer and three alternatives. 

The correct response will exenjplify the proper 
application of the given rules and will reflect the 
appropriate gender a^d number. 

Distractors will consist of the following: 

- a pronoun in the correct case, but incorrect in 
number or gender; 

- a pronoun in the Incorrect case, but correct in 
number and gender; 

- a pronoun representing an Incorrect referent, but 
correct in case, number, and gender; 

With such a set of alternatives, a student's wrong answer choice might . 
provide information on whether he/she was having difficulty in identifying 
referents, was confused about case rules, and/or was having difficulties 
associated with number or gender. 

The result of this specification process is a map for developing test 
items and likewise a map guiding instruction. Not only does the 
specification provide rules for developing multiple parallel test items, it 
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likewise can be used to generate relevant exercises for classroem practice 
practice which will help students to acquire the specific skills they are 
'intended to learn. So, although the process takes some time and effort, 
there are pay-offs. ^ 
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STEP V - REVIEW THE TEST ITEMS 

Once Items are developed, the next step 1s to conduct a thorough 
review, considering two basic questions: j 
Do the Hems matcih their specification? 

Are they free from technical flaws, I.e., do they 
follow conventional rules of Item construction? 

Do the Items match their specification? The answer to this question 
Is critical- to establlsfjlng the content validity of the test. The process 
Is straightforward: have each Item examined by a colleague to compare Its 
match with each element In the domain specification* That Is, the 
description of eligible subject matter and Item features prov1dedj^"the 
content limits need tdf be compared with the content and features of the 
test question; the specification rules for creating correct and incorrect' 
answer alternatives are compared with the actual set provided In selected 
response Items. T^e Items are checked dso to see that they follow the 
prescribed forma/ and that appropriate directions are given. While covered 
agjrtn under "technical flaws," check also to assure that the language us^d 
In the Items Is not unnecessarily difficult or complex and they are free 
from content that might be biased against particular groups of students. 
Where any problems are encountered, suitable Item revisions need to be 
made. In some case there may be Instances where the specification needs to 
be changed Instead. 

AreHhe Items free from technical flaws? The review process here Is 
also straightforward. Check the Items against the general rules for 
constructing test Items of particular types, and where flaws are detected, 
correct them. As with the content review described above, It is 
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preferable to have the review conducted by a colleague (the advantage of 
having a "cold," obj^tlve eye). 
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STEP VI - FIELD TEST THE ITEMS 

F1el4 testing the items is a final step In the test development 
process to assure high quality Items, to verify the test structure, and to 
determine the number of items that will be needed to reliably diagnose 
students' performance. The optimal field test procedures involve a two 
stage process: 1. pilot test the items with a small sample of students to , 
check their appropriateness; 2. administer the test to a larger sample to 
validate the subskills that need to be Included in. the test apd the number 
of Items required for each skill and subskill. 

The purpose of the first pilot test is to determine whether the items 
are appropriate for students and to identify items that are potentially in 
need of revision. Have a small number of stiidents (15-20) who are similar 
to the Intended student popluation take the entire test and provide 
feedback on any problems they encounter, e.g.. vocabulary or directions 
that are unclear; items where there seem to be more than one (or no) right 
answers. This feedback helps indicate where revisions are necessary. 

Item difficulty indices (the percent of students who answer an Item 
correctly) also help signal potential problem Items. Because they are 
based on the same blueprint, one expects similar item difficulties for all 
items measuring the same subskill or task. " For example, item difficulties 
for four of the five Items measuring a particular subskill are .5-. 7; 
however the ^ifflcul ty ' of the fifth item is .25. This latter item should 
be re-examined to determine whether it is, aberrant anti Is unintentionally 
confusing the correct response, whether it matches the specification, 
whether it represents a problem type that is different from the other 
Items, whether the correct answer has b^n miskeyed and/or whether there 
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are typographical or other errors in the items. Any detected errors or 
deviations will need to be corrected* Item difficulties can also be used 
to help Judge the appropriateness of the test for particular students. In 
order to Be useful in a diagncstitrserrsev a test should measure target 
skills which are diffcult for a substantial number of students: If all or 
most students get all or most of the items correct, there is little to 
diagnose. , 
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Troubleshooting: What To Do If, 



You type In a response to 
a computer question or you 
punch a command key such as 
RETURN and nothing happens. 



1. Make sure that you type In re- 
sponse exactly as they are shown 
In this manual . If you add an • 
extra Tetter or puntuatlon mark, 
or leave one out, the computer 
win not understand you. 

2. If you press a response key and 
there is a "beep" sound as well 
as a screen change, then the 
computer has not understood or 
accepted whather you typed in. 
Oust repunchr the key until the 
computer responds with a beep. 



You get an error message 
like: ^ 



1. Press the RETURN key and retype 
your command a second time. Be 
extra careful of spelling, 
spacing and punctuation. Then 
press RETURN. 



You respond to an error message 

and the computer still won't work!... 



Take a deep breath and: 1) turn the 
computer off by using the switch on 
the rear left side; 2) wait a few 
seconds; 3) turn it back on. The 
Menu will automatically reappear. 



You can't find a file, or the 
computer says: 

FILE NOT FOUND 



Make sure that you have not made a 
typographical error on the file 
name.- Retype the correct file name 
(don't forget the DX) and try again. 

If that doesn't work, you may have 
the wrong file name or the student 
may have used a file name that you 
were not expecting. So, you must, 
examine the list of file names: 
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